Traffic light occlusion detection for autonomous vehicle

ABSTRACT

An occlusion detection system for an autonomous vehicle is described herein, where a signal conversion system receives a three-dimensional sensor signal from a sensor system and projects the three-dimensional sensor signal into a two-dimensional range image having a plurality of pixel values that include distance information to objects captured in the range image. A localization system detects a first object in the range image, such as a traffic light, having first distance information and a second object in the range image, such as a foreground object, having second distance information. An occlusion polygon is defined around the second object and the range image is provided to an object perception system that excludes information within the occlusion polygon to determine a configuration of the first object. A directive is output by the object perception system to control the autonomous vehicle based upon occlusion detection.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/124,243, filed on Sep. 7, 2018, and entitled “TRAFFIC LIGHT OCCLUSIONDETECTION FOR AUTONOMOUS VEHICLE”, the entirety of which is incorporatedherein by reference.

BACKGROUND

An autonomous vehicle is a motorized vehicle that can operate without ahuman driver. An exemplary autonomous vehicle includes a plurality ofsensor systems, such as, but not limited to, a camera sensor system, alidar sensor system, a radar sensor system, amongst others, wherein theautonomous vehicle operates based upon sensor signals output by thesensor systems. Typically, sensor signals are provided to a computingsystem in communication with the plurality of sensor systems, whereinthe sensor signals capture objects in proximity to the autonomousvehicle, such as traffic lights and occluding objects. The sensorsignals are processed by the computing system and, based upon detectionof a traffic light or other object captured in the sensor signal, theprocessor executes instructions to control a mechanical system of theautonomous vehicle, such as a vehicle propulsion system, a brakingsystem, or a steering system.

Unlike detection of physical objects, detecting the illuminatedconfiguration of a traffic light is particularly suited for camerasensor systems that generate two-dimensional image sensor signals whichcapture emitted light from the traffic light. The camera sensor systemmay sample a field of view at approximately 10 hertz so that when aconfiguration of the traffic light changes, such as from a red light toa green light, the mechanical systems of the autonomous vehicle can bemanipulated in accordance therewith. The autonomous vehicle is thencontrolled based upon a directive generated according to the detectedconfiguration of the traffic light. For example, a solid red lightcorresponds to a directive of STOP, whereas as a flashing red lightcorresponds to a directive of STOP_AND_YIELD

Problematically for camera sensor systems is that when an occludingobject passes between the traffic light and a lens of the camera, thecomputing system may inaccurately detect the configuration of thetraffic light. For example, if an autonomous vehicle is disposed infront of a solid red light requiring the vehicle to remain stopped, buta tall truck or passing train crosses in front of the autonomous vehicleand temporarily occludes the traffic light from the viewing range of thecamera sensor system, the computing system may detect a flashing redlight instead of the solid red light. Thus, the control system mayerroneously instruct the autonomous vehicle to advance through theintersection, as it otherwise would if the traffic light were a flashingred light.

While detection of depth/distance may be used to distinguish between atraffic light body and an occluding object disposed in front of thetraffic light body, such techniques are complicated by the need forcamera sensor systems that define two-dimensional images which captureemitted light from the traffic light. In addition, while athree-dimensional sensor system, such as a lidar, may supplement acamera sensor system by generating a point cloud that defines variousdistances to objects, such as the traffic light body and an occludingobject, the processing time of three-dimensional sensor signals is muchslower than the processing time of two-dimensional sensor signals,thereby causing a lag in processing times between the sensor signals ofthe two types of sensor systems.

SUMMARY

The following is a brief summary of subject matter that is described ingreater detail herein. This summary is not intended to be limiting as tothe scope of the claims.

Described herein are various technologies pertaining to traffic lightocclusion detection for an autonomous vehicle. With more specificity,described herein are various technologies pertaining to improving thereliability of traffic light detection by generating a sensor signalthat incorporates distance information to detect an occlusion whilemaintaining a processing time comparable to conventional two-dimensionalsensor signals. With still more specificity, described herein is atraffic light detection system that converts a three-dimensional pointcloud into a two-dimensional range image having a plurality of pixelvalues which include distance information to objects captured in therange image. A configuration of a first object (e.g., traffic light) inthe range image is determined by defining an occlusion polygon around asecond object (e.g., a foreground/occluding object) in the range imageand excluding the region of the range image captured by the occlusionpolygon from the configuration determination of the first object,wherein the configuration determination of the first object is basedupon a remaining portion of the range image outside the occlusionpolygon. A mechanical system such as a vehicle propulsion system, abraking system, or a steering system is controlled based upon theconfiguration determination of the first object from the remainingportion of the range image.

An exemplary autonomous vehicle can incorporate various systems toprovide traffic light occlusion detection. Alternatively, the autonomousvehicle can incorporate a single system that performs all of therequired functions for traffic light occlusion detections. A signalconversion system can first receive a three-dimensional sensor signalgenerated by a sensor system. The sensor system may be a lidar cameraand the sensor signal may be a point cloud that captures a field of viewof an environment in which an autonomous vehicle is located; forinstance, one or more objects may be in the field of view. The signalconversion system projects the three-dimensional sensor signal (pointcloud) into a two-dimensional range image/pixel patch, wherein eachpixel in the range image includes a pixel value that incorporatesdistance information to objects disposed at a corresponding point in therange image. Distances from the sensor system that are too remote fordetection (e.g., outside the detection range of the particularlidar/sensor system attached to the autonomous vehicle) are given apixel value of undefined. Thus, each pixel value in the range image iseither defined or undefined and, if defined, the distance informationidentifies a distance from the sensor system to an object captured atthat pixel.

A localization system can further receive the range image output by thesignal conversion system. The localization system is configured tocalibrate and localize objects captured in the generated range imagebased upon a geometric projection between the sensor system and adetected object, such as a traffic light. The localization system maydefine a region of interest that approximately centralizes a detectedtraffic light therein and is typically dimensioned to be larger than thedimensions of a traffic light to provide a tolerance for map calibrationand localization errors. For example, the region of interest may besized such that a 1-meter spatial buffer would be provided around theperimeter of a centralized traffic light. The range image/pixel patch isthen cropped by the localization system to define the region of interestaccording to desirable dimensions.

The region of interest includes the traffic light as well as distanceinformation corresponding thereto. Distance information of pixel valuesfor the entire region of interest is separated into three groups: 1)distances to objects that are roughly the same as the distanceinformation corresponding to the traffic light (e.g., the body of thetraffic light, the pole or wire supporting the traffic light, etc.); 2)distances to objects that are closer than the distance informationcorresponding to the traffic light; and 3) distances to objects that arefarther than the distance information corresponding to the trafficlight. Closer distance information may correspond to occludingforeground objects, whereas farther distance information corresponds tobackground points. While an occlusion polygon is defined aroundforeground objects in the region of interest to identify occlusions totraffic lights, the traffic light detection architecture is configuredto identify and remove detected background lights by defining a similarpolygon around such objects within the region of interest. The occlusionpolygon and/or related background polygon is typically configured as aconvex polygon.

An object perception system can additionally receive the region ofinterest capturing the traffic light, which may include the occlusionpolygon defined around a foreground object. The object perception systemlikewise comprises an occlusion reasoning module configured to track,store, and predict the position of a traffic light based upon positivetraffic light detections. Thus, when the foreground object/occlusionpolygon occupies the same spatial position in the region of interest asthe traffic light, the occlusion reasoning module can determine that thetraffic light is occluded. Full occlusion occurs when the foregroundobject completely covers the detected location of the traffic light,whereas partial occlusion occurs when the foreground object blocks onlya portion of the detected traffic light. The occlusion reasoning modulemay also detect and exclude emitted bursts of light from traffic lightconfiguration determinations. For example, an object detector may detectan extraneous burst of yellow light, but the occlusion reasoning modulemay confirm that the light is still green and exclude the detected burstof yellow light.

Accordingly, the occlusion reasoning module can determine theconfiguration of the traffic light by sampling range images at aparticular sampling rate and excluding samples where the traffic lightwas occluded or otherwise obscured to generate a directive based upononly non-occluded/non-obscured samples. In some instances, the occlusionreasoning module may also determine the configuration of a detectedtraffic light from partially occluded range image samples. For example,if the position of a green light is occluded but a red light is detectedand not occluded, the occlusion reasoning module may include the samplein a traffic light configuration determination regarding red lightdirectives. Alternatively, if the red light is no longer illuminated,the occlusion reasoning module could make a configuration determinationregarding an occluded bulb that is illuminated (e.g., reasoning that anoccluded green light is illuminated when non-occluded red and yellowlights are not illuminated).

The above summary presents a simplified summary in order to provide abasic understanding of some aspects of the systems and/or methodsdiscussed herein. This summary is not an extensive overview of thesystems and/or methods discussed herein. It is not intended to identifykey/critical elements or to delineate the scope of such systems and/ormethods. Its sole purpose is to present some concepts in a simplifiedform as a prelude to the more detailed description that is presentedlater.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary autonomous vehicle.

FIG. 2 illustrates an exemplary architecture configured to output adirective for an autonomous vehicle based upon detection of a trafficlight occlusion.

FIG. 3 illustrates an exemplary architecture configured to output adirective for an autonomous vehicle based upon detection of a trafficlight occlusion.

FIG. 4 illustrates a same field of view at three sequential timeframes.

FIG. 5 is a flow diagram illustrating an exemplary methodology fortraffic light occlusion detection for an autonomous vehicle.

FIG. 6 is a flow diagram illustrating an exemplary methodology fortraffic light occlusion detection for an autonomous vehicle.

FIG. 7 illustrates an exemplary computing system.

DETAILED DESCRIPTION

Various technologies pertaining to traffic light occlusion detection foran autonomous vehicle is now described with reference to the drawings,wherein like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of one or more aspects. It may be evident, however, thatsuch aspect(s) may be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate describing one or more aspects. Further, itis to be understood that functionality that is described as beingcarried out by certain system components may be performed by multiplecomponents. Similarly, for instance, a component may be configured toperform functionality that is described as being carried out by multiplecomponents.

Moreover, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B.

In addition, the articles “a” and “an” as used in this application andthe appended claims should generally be construed to mean “one or more”unless specified otherwise or clear from the context to be directed to asingular form.

Further, as used herein, the terms “component”, “module”, and “system”are intended to encompass computer-readable data storage that isconfigured with computer-executable instructions that cause certainfunctionality to be performed when executed by a processor. Thecomputer-executable instructions may include a routine, a function, orthe like. It is also to be understood that a component, module, orsystem may be localized on a single device or distributed across severaldevices.

Further, as used herein, the term “exemplary” is intended to meanserving as an illustration or example of something and is not intendedto indicate a preference.

As used herein, the term “occlusion” refers to an object that isspatially positioned such that it partially or completely obstructssensor system detection of emitted light from a traffic light. Asfurther used herein an “occlusion polygon” defines an imaginaryperimeter around an occluding object.

With reference now to FIG. 1 , an exemplary autonomous vehicle 100 isillustrated. The autonomous vehicle 100 can navigate about roadwayswithout a human driver based upon sensor signals output by sensorsystems 102-104 of the autonomous vehicle 100. The autonomous vehicle100 includes a plurality of sensor systems 102-104 (a first sensorsystem 102 through an Nth sensor system 104). The sensor systems 102-104are of different types and are arranged about the autonomous vehicle100. For example, the first sensor system 102 may be a camera sensorsystem and the Nth sensor system 104 may be a lidar system; or thesensor systems 102-104 may be of different types of a same kind ofsensor system, such as different types of camera sensor systems (e.g.,fixed exposure and autoexposure). Other exemplary sensor systems includeradar sensor systems, global positioning system (GPS) sensor systems,infrared sensor systems, sonar sensor systems, and the like.

While certain sensor systems have limited individual viewing ranges(e.g., a camera sensor system may have a viewing range of 66 degrees),the incorporation of additional sensor systems 102-104 to the autonomousvehicle 100 can increase the viewing range of the sensor systems 102-104up to 180 degrees and beyond, if desirable. Furthermore, some or all ofthe plurality of sensor systems 102-104 may comprise articulatingsensors. An articulating sensor is a sensor that may be oriented (i.e.,rotated) by the autonomous vehicle 100 such that a field of view of thearticulating sensor may be directed towards different regionssurrounding the autonomous vehicle 100.

The autonomous vehicle 100 further includes several mechanical systemsthat are used to effectuate appropriate motion of the autonomous vehicle100. For instance, the mechanical systems can include but are notlimited to, a vehicle propulsion system 106, a braking system 108, and asteering system 110. The vehicle propulsion system 106 may include anelectric motor, an internal combustion engine, or both. The brakingsystem 108 can include an engine break, brake pads, actuators, and/orany other suitable componentry that is configured to assist indecelerating the autonomous vehicle 100. The steering system 110includes suitable componentry that is configured to control thedirection of movement of the autonomous vehicle 100 during propulsion.

The autonomous vehicle 100 additionally comprises a computing system 112that is in communication with the sensor systems 102-104 and is furtherin communication with the vehicle propulsion system 106, the brakingsystem 108, and the steering system 110. The computing system 112includes a processor 114 and memory 116 that includescomputer-executable instructions that are executed by the processor 114.In an example, the processor 114 can be or include a graphics processingunit (GPU), a plurality of GPUs, a central processing unit (CPU), aplurality of CPUs, an application-specific integrated circuit (ASIC), amicrocontroller, a programmable logic controller (PLC), a fieldprogrammable gate array (FPGA), or the like.

The memory 116 comprises three systems which may cooperate to providetraffic light occlusion detection (e.g., a signal conversion system 118,a localization system 120, and an object perception system 122). Thesignal conversion system 118 is configured to receive a sensor signal,such as a point cloud, that represents a three-dimensional environmentfrom a sensor system and project the three-dimensional point cloud intoa two-dimensional range image having distance information thatdifferentiates foreground objects from focal objects such as trafficlights. The localization system 120 detects objects captured in therange image based upon the distance information and defines an occlusionpolygon around foreground objects, which are excluded from aconfiguration determination of the focal object (traffic light). Theobject perception system 122 receives the range image from thelocalization system 120 and generates a directive for controlling theautonomous vehicle 100 based upon a configuration determination of thefocal object. The configuration determination may or may not be basedupon exclusion of the occlusion polygon from the range image.

The memory 116 further includes a control system 124 configured toreceive the directive output by the object perception system 122 andcontrol at least one mechanical system (e.g., the vehicle propulsionsystem 106, the brake system 108, and/or the steering system 110) of theautonomous vehicle 100 in accordance therewith. The generated directivedefines the most suitable course of action that an autonomous vehicle100 should perform according to the detected configuration of thetraffic light and/or lane as well as the applicable laws of the region.

With reference now to FIG. 2 , an architecture 200 is implemented by thecomputing system 112 to output a directive for an autonomous vehiclebased upon detection of a traffic light occlusion captured in a sensorsignal generated by the sensor systems 102-104. The sensor systems102-104 may include a plurality of lidar cameras that provide a firstsensor signal and a second sensor signal (e.g., a first point cloud anda second point cloud) to the signal conversion system 118. Additionally,the sensor signals may be generated by sensor systems 102-104 ofdifferent types where, for example, the first sensor signal can be apoint cloud and the second sensor signal is a camera image correspondingto a same field of view captured by the first sensor signal. Theinformation captured in the sensor signals is provided to the signalconversion system 118 and further to the localization and objectperception systems 120-122. Thus, the architecture 200 illustrates amultimodal system configured to generate a directive 208 based upon aplurality of sensor signal inputs.

The signal conversion system 118 is configured to receive athree-dimensional sensor signal from the sensor systems 102-104 andconvert the three-dimensional sensor signal into a two-dimensional rangeimage/sparse depth map. The range image comprises a plurality of pixelvalues that define distance information to one or more objects capturedin a field of view of the range image. In particular, the distanceinformation of each pixel defines a distance at each point in the rangeimage. Features in the field of view that are disposed far away from thesensor systems 102-104, such as a distant object or the sky, have acorresponding pixel value of undefined and are not identified in therange image by finite distance values but do correspond to distanceinformation, albeit undefined values. The signal conversion system 118may also receive a two-dimensional camera image that is approximatelythe same size as a three-dimensional lidar point cloud to improveconversion of the point cloud into a range image. Since the processingtime of three-dimensional sensor signals can be slower than theprocessing time of two-dimensional sensor signals, conversion of thepoint cloud into a range image by the conversion system 118 resolvesconventional processing drawbacks associated with usingthree-dimensional sensor systems to detect occlusions.

The localization system 120 receives the range image output by thesignal conversion system 118 and includes a traffic light copilot module202, a region of interest module 204, and an occlusion polygon module206. In an exemplary embodiment, the region of interest module 204 andthe occlusion polygon module 206 may be incorporated within the trafficlight copilot module 202. The region of interest module 204 and theocclusion polygon module 206 may also be included together as featuresof a single calibration module. The traffic light copilot module 202defines a geometric projection that identifies where an object, such asa traffic light, is positioned relative to the sensor systems 102-104 ofthe autonomous vehicle 100. The output of the traffic light copilotmodule 202 is provided to the occlusion polygon module 206 and/or theregion of interest module 204. The region of interest module 204 definesa region of interest that frames a focal object, such as a trafficlight, captured in the range image. The region of interest comprisesdimensions that are larger than the focal object to provide a tolerancefor map calibration and localization errors (e.g., the region ofinterest may correspond to a height of 3 meters when defined around atraffic light having a height of 1 meter).

The occlusion polygon module 206 identifies foregroundobjects/occlusions by defining an occlusion polygon around theforeground objects based upon the distance information incorporated inthe range image. If a region of interest is defined by the region ofinterest module 204, the occlusion polygon module 206 generally definesthe occlusion polygon around a foreground object within the region ofinterest of the focal object. Otherwise, the occlusion polygon isdefined around a foreground object in the range image that is inproximity to the focal object. While the occlusion polygon is typicallyformed as a convex polygon, it is not limited to any specificconfiguration. When a region of interest is defined around a focalobject, the range image is cropped to the dimensions of the region ofinterest and provided to an object perception system 122, along with anyocclusion polygon or remaining portions thereof defined around aforeground object within the region of interest. The architecture 200outputs a directive 208 from the object perception system 122 to controlthe autonomous vehicle 100 based upon detection of an occluded focalobject.

With reference now to FIG. 3 , architecture 300 is implemented by thecomputing system 112 to output a directive 208. The object perceptionsystem 122 of architecture 300 includes an object detector 302 having anocclusion reasoning module 306. The object detector 302 detects theconfiguration of a traffic light captured in an input range image togenerate the directive 208 based upon illuminated bulbs thereof. Theobject detectors 302 can be, but are not limited to, one of an absolutedetector which detects a kind of bulb that is illuminated (e.g., a solidred circle) or a relative activation detector which determines theconfiguration of a traffic light based upon inferences about the layoutof the traffic light. For example, if the top position of a verticallyaligned three-bulb traffic light is illuminated, the relative activationdetector may infer a “solid red circle” based upon a taxonomy ofpredefined layouts incorporated in the memory 116 to generate thedirective 208.

The occlusion reasoning module 306 confirms or rejects the detectedconfiguration of the traffic light by the object detector 302 based uponwhether the range image includes an occlusion that is fully or partiallyblocking the traffic light from sensor signals emitted by the sensorsystems 102-104. If the configuration of the traffic light cannot beconfirmed in a particular range image, such as when an emitted burst oflight obscures object detection, the range image may be excluded by theocclusion reasoning module 306 from traffic light configurationdeterminations.

The occlusion reasoning module 306 may further sample a plurality ofrange images to track, store, and predict the configuration of a trafficlight detected by the localization system 120. In range image samplesthat include an occlusion to the traffic light, the occlusion reasoningmodule 306 may exclude such samples from a configuration determinationor it may define an expected configuration based upon informationdetected from one or more previous samples. For example, a previousrange image may capture the vertically aligned three-bulb traffic lighthaving an illuminated solid red circle but in a current range image thetraffic light is fully occluded. The occlusion reasoning module 306 candetermine based upon the layout of the traffic light that such lights donot generally change from solid red to flashing red during brief periodsof occlusion. Additionally, the occlusion reasoning module 306 maydetermine that an occluded traffic light is expected to change from asolid red circle to a solid green circle after a certain length of time.Thus, if only the bottom (green bulb) of the traffic light is occludedbut the red light and yellow light is exposed to the sensor systems102-104, the occlusion reasoning module 306 can determine that a solidgreen circle is illumined when the solid red/yellow circles are nolonger in a light emitting configuration.

When a plurality of range images is provided to the localization system120 and further to the object perception system 122, the objectdetectors 302 generate a plurality of independent directives 308 thatcorresponds to the plurality of range images. Each independent directive308 defines a (pre-fusion) vehicle maneuver according to a configurationof the traffic light detected by individual object detectors 308 that isbased upon occlusion reasoning with respect to a specific range imageinput from the localization system 120. The plurality of independentdirectives 308 are subsequently provided to a signal fusion module 310where they are fused together to output a fused directive 208 forcontrolling the autonomous vehicle 100. If only one range image isprovided to the object perception system 122, or if a plurality ofobject detector inputs is fused into a single directive at the objectdetector level, fusion is not required by the signal fusion module 310and the independent directive 308 is itself output as the fuseddirective 208. The directive 208 is then provided to the control system124 for controlling the autonomous vehicle 100.

Further included in the architecture 300 is a convolution neural network304 and a directive state machine 312. The convolution neural network304 is linked to the object detector 302 to identify objectconfigurations in the region of interest/range image that are output bythe localization system 120. In an exemplary embodiment, a plurality ofconvolution neural networks 304 can be executed on a same input (rangeimage) to the object perception system 122 to detect a plurality ofobject configurations captured in the range image. The directive statemachine 312 is in communication with the object perception system 122.The directive state machine 312 is configured to define at least eightuniversal directives including: STOP (red light), STOP_AND_YIELD(flashing red light), MAYBE_STOP (yellow light), YIELD (flashing yellowlight), ABOUT_TO_GO (light will soon turn green—transition directive insome countries), GO (green light), GO_PROTECTED (proceed through), andUNKNOWN (no detected light).

With reference now to FIG. 4 , a same field of view is illustrated atthree sequential timeframes 1), 2), and 3). The field of view includes aseries of trucks 402 passing through an intersection directed by atraffic light 404 having an illuminated solid red circle 406. A regionof interest 408 frames the traffic light 404 and is provided to theobject perception system 122 to determine a configuration thereof. Theregion of interest 408 corresponding to each timeframe 1), 2), and 3)includes an occlusion polygon 410 defined around a foreground object tothe traffic light 404 based upon distance information incorporated inthe range image.

At timeframe 1), the occlusion polygon 410 captures a cab portion of thetruck 402. However, the traffic light 404 is not occluded within theregion of interest 408. At timeframe 2), the truck 402 has advancedforward such that an enclosed trailer of the truck 402 is now fullyoccluding the traffic light 404. The occlusion polygon 410 is configuredin the region of interest 408 of timeframe 2) to capture the enclosedtrailer portion of the truck 402 where the traffic light 404 wouldotherwise be disposed. At timeframe 3), the truck 402 has advance evenfarther forward so that the traffic light 404 is once again exposedwithin the region of interest 408. The region of interest 408 oftimeframe 3) includes a plurality of occlusion polygons 410 that capturea rear of the first truck and a front of the second truck. Illustratinga fourth and fifth timeframes from the same field of view would furtherperpetuate the pattern of occlusion followed by non-occlusion of thetraffic light 404 due to the second truck 402.

Without the occlusion polygons 410 to differentiate between foregroundobjects and the traffic light 404, the object perception system 122 may,in two-dimensional object detection, determine that the bulb is aflashing red circle as opposed to a solid red circle 406 due to briefocclusion caused by the passing trucks 402. Accordingly, the occlusionreasoning module 306 is configured to detect an occlusion via theocclusion polygon 410 and exclude a range image sample in accordancetherewith. Specifically, timeframe 2) may be exclude by the occlusionreasoning module 306 so that a configuration of the traffic light 404 isdetermined based upon only timeframes 1) and 3). This exclusion reducesthe likelihood that a flashing red circle will be erroneously detectedinstead of the solid red circle 406.

FIGS. 5 and 6 illustrate exemplary methodologies relating to trafficlight occlusion detection for an autonomous vehicle. While themethodologies are shown and described as being a series of acts that areperformed in a sequence, it is to be understood and appreciated that themethodologies are not limited by the order of the sequence. For example,some acts can occur in a different order than what is described herein.In addition, an act can occur concurrently with another act. Further, insome instances, not all acts may be required to implement a methodologydescribed herein.

Moreover, the acts described herein may be computer-executableinstructions that can be implemented by one or more processors and/orstored on a computer-readable medium or media. The computer-executableinstructions can include a routine, a sub-routine, programs, a thread ofexecution, and/or the like. Still further, results of acts of themethodologies can be stored in a computer-readable medium, displayed ona display device, and/or the like.

Referring now to FIG. 5 , an exemplary methodology 500 for traffic lightocclusion detection for an autonomous vehicle is illustrated. Themethodology 500 starts at 502, and at 504 a sensor signal is generatedby a sensor system. The sensor signal may be a point cloud and thesensor system may be a lidar camera. At 506, the sensor signal isprojected to a range image having a plurality of pixel values thatinclude distance information, wherein the distance information of eachpixel value defines a distance to an object captured in the range image.In an exemplary methodology, the sensor signal is received from thesensor system by a conversion system that projects the sensor signal tothe range image. At 508, a first object is detected having firstdistance information and a second object is detected having seconddistance information. The first distance information typically defines afirst distance that is longer than a second distance defined by thesecond distance information, although the methodology 500 is not limitedto this configuration. At 510, an occlusion polygon is defined aroundthe second object by the localization system based upon the seconddistance information. The occlusion polygon may further be included in aregion of interest which is defined by the localization system to framethe first object. At 512, a portion of the range image identified by theocclusion polygon is excluded from a configuration determination of thefirst object. At 514, a mechanical system is controlled by a controlsystem based upon the configuration determination of the first object.The methodology 500 completes at 516.

Referring now to FIG. 6 , an exemplary methodology 600 for traffic lightocclusion detection for an autonomous vehicle is illustrated. Themethodology 600 starts at 602, and at 604 a plurality of range images issampled by an occlusion reasoning module to detect an object. Theplurality of range images has a plurality of pixel values that eachinclude distance information to an object captured in the range image.At 606, a first object having first distance information and a secondobject having second distance information are detected from theplurality of range images. At 608, a sampled range image is excludedfrom a configuration determination of the first object by the occlusionreasoning module when the second object occludes the first object in thesampled range image. A sampled range image may also be excluded from aconfiguration determination of the first object when an emitted burst oflight is detected in the sampled range image by the occlusion reasoningmodule. At 610, the configuration of the first object is determined bythe occlusion reasoning module based upon non-excluded range imagesamples. At 612, a mechanical system is controlled by a control systembased upon the configuration determination of the first object. Themethodology 600 completes at 614.

Referring now to FIG. 7 , a high-level illustration of an exemplarycomputing device 700 that can be used in accordance with the systems andmethodologies disclosed herein is illustrated. For instance, thecomputing device 700 may be or include the computing system 112. Thecomputing device 700 includes at least one processor 702 that executesinstructions that are stored in a memory 704. The instructions may be,for instance, instructions for implementing functionality described asbeing carried out by one or more modules and systems discussed above orinstructions for implementing one or more of the methods describedabove. In addition to storing executable instructions, the memory 704may also store location information, distance information, directioninformation, etc.

The computing device 700 additionally includes a data store 708 that isaccessible by the processor 702 by way of the system bus 706. The datastore 708 may include executable instructions, location information,distance information, direction information, etc. The computing device700 also includes an input interface 710 that allows external devices tocommunicate with the computing device 700. For instance, the inputinterface 710 may be used to receive instructions from an externalcomputer device, etc. The computing device 700 also includes an outputinterface 712 that interfaces the computing device 700 with one or moreexternal devices. For example, the computing device 700 may transmitcontrol signals to the vehicle propulsion system 106, the braking system108, and/or the steering system 110 by way of the output interface 712.

Additionally, while illustrated as a single system, it is to beunderstood that the computing device 700 may be a distributed system.Thus, for instance, several devices may be in communication by way of anetwork connection and may collectively perform tasks described as beingperformed by the computing device 700.

Various functions described herein can be implemented in hardware,software, or any combination thereof. If implemented in software, thefunctions can be stored on or transmitted over as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia includes computer-readable storage media. A computer-readablestorage media can be any available storage media that can be accessed bya computer. By way of example, and not limitation, suchcomputer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that can be used to store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. Disk and disc, as used herein, includecompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproducedata magnetically and discs usually reproduce data optically withlasers. Further, a propagated signal is not included within the scope ofcomputer-readable storage media. Computer-readable media also includescommunication media including any medium that facilitates transfer of acomputer program from one place to another. A connection, for instance,can be a communication medium. For example, if the software istransmitted from a website, server, or other remote source using acoaxial cable, fiber optic cable, twisted pair, digital subscriber line(DSL), or wireless technologies such as infrared, radio, and microwave,then the coaxial cable, fiber optic cable, twisted pair, DSL, orwireless technologies such as infrared, radio and microwave are includedin the definition of communication medium. Combinations of the aboveshould also be included within the scope of computer-readable media.

Alternatively, or in addition, the functionally described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Application-specific Integrated Circuits (ASICs),Application-specific Standard Products (ASSPs), System-on-a-chip systems(SOCs), Complex Programmable Logic Devices (CPLDs), etc.

What has been described above includes examples of one or moreembodiments. It is, of course, not possible to describe everyconceivable modification and alteration of the above devices ormethodologies for purposes of describing the aforementioned aspects, butone of ordinary skill in the art can recognize that many furthermodifications and permutations of various aspects are possible.Accordingly, the described aspects are intended to embrace all suchalterations, modifications, and variations that fall within the spiritand scope of the appended claims. Furthermore, to the extent that theterm “includes” is used in either the detailed description or theclaims, such term is intended to be inclusive in a manner similar to theterm “comprising” as “comprising” is interpreted when employed as atransitional word in a claim.

What is claimed is:
 1. An autonomous vehicle, comprising: a sensorsystem; a camera system; and a computing system in communication withthe sensor system and the camera system, wherein the computing systemcomprises: a processor; and memory that stores computer-executableinstructions that, when executed by the processor, cause the processorto perform acts comprising: converting a three-dimensional point cloudoutputted by the sensor system to a two-dimensional range image, thetwo-dimensional range image comprises distance values to objects in anenvironment nearby the autonomous vehicle; detecting a first region inthe two-dimensional range image having distance values corresponding toan expected distance to a traffic light and a second region in thetwo-dimensional range image having distance values farther than theexpected distance to the traffic light; defining a polygon around thesecond region; identifying an illuminated configuration of the trafficlight based on an output from the camera system, wherein a region of theoutput from the camera system corresponding to the polygon is excludedfrom being used to identify the illuminated configuration of the trafficlight; and controlling the autonomous vehicle based on the illuminatedconfiguration of the traffic light.
 2. The autonomous vehicle of claim1, wherein a light is detected in the region of the output from thecamera system corresponding to the polygon and excluded from being usedto identify the illuminated configuration of the traffic light.
 3. Theautonomous vehicle of claim 1, wherein the memory further storescomputer-executable instructions that, when executed by the processor,cause the processor to perform acts comprising: defining a region ofinterest in the two-dimensional range image that frames the trafficlight; wherein the polygon is within the region of interest.
 4. Theautonomous vehicle of claim 3, wherein distance values of pixels withinthe region of interest are evaluated to detect the first region and thesecond region.
 5. The autonomous vehicle of claim 1, wherein the polygonis a convex polygon.
 6. The autonomous vehicle of claim 1, wherein thesensor system is a lidar sensor system that generates thethree-dimensional point cloud.
 7. The autonomous vehicle of claim 1,wherein the memory further stores computer-executable instructions that,when executed by the processor, cause the processor to perform actscomprising: detecting a third region in the two-dimensional range imagehaving distance values closer than the expected distance to the trafficlight; and defining a differing polygon around the third region; whereina differing region of the output from the camera system corresponding tothe differing polygon is excluded from being used to identify theilluminated configuration of the traffic light.
 8. The autonomousvehicle of claim 1, wherein the memory further stores instructions that,when executed by the processor, cause the processor to perform actscomprising: converting a differing three-dimensional point cloudoutputted by the sensor system for a differing time period to adiffering two-dimensional range image; detecting a third region in thediffering two-dimensional range image having distance valuescorresponding to the expected distance to the traffic light and a fourthregion in the differing two-dimensional range image having distancevalues farther than the expected distance to the traffic light; defininga differing polygon around the fourth region; wherein the illuminatedconfiguration of the traffic light is further identified based on adiffering output from the camera system for the differing time period,and wherein a region of the differing output of the camera systemcorresponding to the differing polygon is excluded from being used toidentify the illuminated configuration of the traffic light.
 9. Theautonomous vehicle of claim 1, wherein the illuminated configuration ofthe traffic light is further identified based on at least a portion of adiffering output from the camera system for a differing time period. 10.A method performed by an autonomous vehicle, comprising: converting athree-dimensional point cloud outputted by a sensor system of theautonomous vehicle to a two-dimensional range image, the two-dimensionalrange image comprises distance values to objects in an environmentnearby the autonomous vehicle; detecting a first region in thetwo-dimensional range image having distance values corresponding to anexpected distance to a traffic light and a second region in thetwo-dimensional range image having distance values farther than theexpected distance to the traffic light; defining a polygon around thesecond region; identifying an illuminated configuration of the trafficlight based on an output from a camera system of the autonomous vehicle,wherein a region of the output from the camera system corresponding tothe polygon is excluded from being used to identify the illuminatedconfiguration of the traffic light; and controlling the autonomousvehicle based on the illuminated configuration of the traffic light. 11.The method of claim 10, wherein a light is detected in the region of theoutput from the camera system corresponding to the polygon and excludedfrom being used to identify the illuminated configuration of the trafficlight.
 12. The method of claim 10, further comprising: defining a regionof interest in the two-dimensional range image that frames the trafficlight; wherein the polygon is within the region of interest.
 13. Themethod of claim 12, wherein distance values of pixels within the regionof interest are evaluated to detect the first region and the secondregion.
 14. The method of claim 10, further comprising: detecting athird region in the two-dimensional range image having distance valuescloser than the expected distance to the traffic light; and defining adiffering polygon around the third region; wherein a differing region ofthe output from the camera system corresponding to the differing polygonis excluded from being used to identify the illuminated configuration ofthe traffic light.
 15. The method of claim 10, further comprising:converting a differing three-dimensional point cloud outputted by thesensor system for a differing time period to a differing two-dimensionalrange image; detecting a third region in the differing two-dimensionalrange image having distance values corresponding to the expecteddistance to the traffic light and a fourth region in the differingtwo-dimensional range image having distance values farther than theexpected distance to the traffic light; defining a differing polygonaround the fourth region; wherein the illuminated configuration of thetraffic light is further identified based on a differing output from thecamera system for the differing time period, and wherein a region of thediffering output of the camera system corresponding to the differingpolygon is excluded from being used to identify the illuminatedconfiguration of the traffic light.
 16. The method of claim 10, whereinthe illuminated configuration of the traffic light is further identifiedbased on at least a portion of a differing output from the camera systemfor a differing time period.
 17. A computing system of an autonomousvehicle, comprising: a processor; and memory that storescomputer-executable instructions that, when executed by the processor,cause the processor to perform acts comprising: converting athree-dimensional point cloud outputted by a sensor system of theautonomous vehicle to a two-dimensional range image, the two-dimensionalrange image comprises distance values to objects in an environmentnearby the autonomous vehicle; detecting a first region in thetwo-dimensional range image having distance values corresponding to anexpected distance to a traffic light and a second region in thetwo-dimensional range image having distance values farther than theexpected distance to the traffic light; defining a polygon around thesecond region; identifying an illuminated configuration of the trafficlight based on an output from a camera system of the autonomous vehicle,wherein a region of the output from the camera system corresponding tothe polygon is excluded from being used to identify the illuminatedconfiguration of the traffic light; and controlling the autonomousvehicle based on the illuminated configuration of the traffic light. 18.The computing system of claim 17, wherein the memory further storescomputer-executable instructions that, when executed by the processor,cause the processor to perform acts comprising: detecting a third regionin the two-dimensional range image having distance values closer thanthe expected distance to the traffic light; and defining a differingpolygon around the third region; wherein a differing region of theoutput from the camera system corresponding to the differing polygon isexcluded from being used to identify the illuminated configuration ofthe traffic light.
 19. The computing system of claim 17, wherein a lightis detected in the region of the output from the camera systemcorresponding to the polygon and excluded from being used to identifythe illuminated configuration of the traffic light.
 20. The computingsystem of claim 17, wherein the illuminated configuration of the trafficlight is further identified based on at least a portion of a differingoutput from the camera system for a differing time period.